Controlling a 2d screen interface application in a mixed reality application

ABSTRACT

The present invention relates to controlling a 2D application in a mixed reality application. In order to provide further simplification in terms of operating devices and equipment for the medical personnel, a device (10) for controlling a 2D application in a mixed reality application is provided. The device comprises a 2D application interface (12), a processor (14), a mixed reality application interface (16) and a control interface (18). The 2D application interface receives data representative of a 2D application. The application comprises interactive elements providing commands for user control of the 2D application. The processor identifies and extracts the interactive elements based on the data representative of the 2D application. The processor also translates the interactive elements to user interface features of a mixed reality display application. The processor further generates data for presenting the mixed reality display application and maps user input to commands for user control of the 2D application. The mixed reality application interface provides the mixed reality display application to a user as mixed reality. The control interface receives input from the user for the user interface features and provides the commands to the 2D application in order to control the 2D application. Hence, augmented reality interfaces are mapped to traditional 2D screen interfaces. This allows to integrate applications that only provided the interaction on a monitor with a mouse or the like to be operated in today&#39;s augmented reality environment.

FIELD OF THE INVENTION

The present invention relates to a device for controlling a 2Dapplication in a mixed reality application, to a system for controllinga 2D application in a mixed reality application and to a method forcontrolling a 2D application in a mixed reality application.

BACKGROUND OF THE INVENTION

In medical environment, wearable augmented reality enables newopportunities for user interfaces as compared to traditional 2Dinterfaces. However, those augmented reality interfaces require specificinputs and design requirements in order to be intuitive compared toprevious 2D interfaces. It has been shown that traditional 2D interfacesdo not integrate into augmented reality tools. This adds to anincreasing complexity in the operational room. It also adds tocomplexity in the users' operational tasks.

SUMMARY OF THE INVENTION

There may thus be a need to provide further simplification in terms ofoperating devices and equipment for the medical personnel.

The object of the present invention is solved by the subject-matter ofthe independent claims; further embodiments are incorporated in thedependent claims. It should be noted that the following describedaspects of the invention apply also for the device for controlling a 2Dapplication in a mixed reality application, for the system forcontrolling a 2D application in a mixed reality application and for themethod for controlling a 2D application in a mixed reality application.

According to the present invention, a device for controlling a 2Dapplication in a mixed reality application is provided. The devicecomprises a 2D application interface, a processor, a mixed realityapplication interface and a control interface. The 2D applicationinterface is configured to receive data representative of a 2Dapplication. The 2D application comprises interactive elements providingcommands for user control of the 2D application. The processor isconfigured to identify and extract the interactive elements based on thedata representative of the 2D application. The processor is alsoconfigured to translate the interactive elements to user interfacefeatures of a mixed reality display application. The processor isfurther configured to generate data for presenting the mixed realitydisplay application. The processor is still further configured to mapuser input to commands for user control of the 2D application. The mixedreality application interface is configured to provide the mixed realitydisplay application to a user as mixed reality. The control interface isconfigured to receive input from the user for the user interfacefeatures. The control interface is also configured to provide thecommands to the 2D application in order to control the 2D application.

As a result, it is possible, for example, to stream a 2D screen-basedinterface into an augmented reality environment while augmented realityspecific inputs are based on extracted features from the 2D interface.User interface information is extracted from the 2D interface (including3^(rd) party applications) for use in creating enhanced interactionwithin augmented reality applications.

According to an example, the 2D application interface is configured toreceive display data for presenting the 2D application on a monitor, ora sequence of images of a camera acquiring image data of a presentationof the 2D application.

According to an example, for the identifying and extracting, theprocessor is configured to provide an interactive elementsidentification routine comprising at least one of the group of:detecting interaction features based on image processing of the 2Dapplication, detecting changes when moving a position indicator acrossthe 2D application presentation, recording locations of interactionactivity by the user in the 2D application presentation and providingtraining datasets comprising interaction activity and recognizinginteractive elements based on machine learning of the training datasets.

According to an example, the processor is configured to create collidersaround the identified interactive elements for the translating of theinteractive elements to the user interface features. The collidersenable a number of 3D feedback features.

According to the present invention, also a system for controlling a 2Dapplication in a mixed reality application is provided. The systemcomprises a device for controlling a 2D application in a mixed realityapplication according to one of the preceding examples. The system alsocomprises a mixed reality presentation arrangement, which isdata-connected to the device for controlling a 2D application in a mixedreality application. The mixed reality presentation arrangementcomprises a presentation device that is configured to present the mixedreality display application to a user as mixed reality. The mixedreality presentation arrangement further comprises a user input device.

According to an example, the mixed reality presentation arrangement isan augmented reality presentation arrangement that comprises a wearableaugmented reality device. In an option, provided in addition oralternatively, a head-mounted display is provided configured to presentthe mixed reality display application as an overlay to the reality.

According to an example, the mixed reality presentation arrangement is amixed reality presentation arrangement that comprises a wearable mixedreality device. In an option, provided in addition or alternatively, ahead-mounted display is provided configured to present the virtualreality display application.

According to an example, the mixed reality presentation arrangement is avirtual reality presentation arrangement that comprises a wearablevirtual reality device. In an option, provided in addition oralternatively, a head-mounted display is provided configured to presentthe virtual reality display application.

According to an example, a monitor is provided configured to present the2D application. In an option, provided in addition or alternatively, aninterface is provided to forward user interaction data to the 2Dapplication. The user interaction data is provided by the user inputdevice of the mixed reality presentation arrangement.

According to the present invention, also a method for controlling a 2Dapplication in a mixed reality application is provided. The methodcomprises the following steps:

-   -   receiving data representative of a 2D application, wherein the        2D graphical user interface comprises interactive elements        providing commands for user control of the 2D application;    -   identifying and extracting the interactive elements based on the        data representative of the 2D application;    -   translating the interactive elements to user interface features        of a mixed reality display application;    -   generating and providing data for presenting the mixed reality        display application to a user as mixed reality, wherein the        mixed reality display application comprises a presentation of        the 2D application;    -   receiving input from the user for the user interface features;        mapping the user input to commands for user control of the 2D        application; and    -   providing the commands to the 2D application in order to control        the 2D application.

In an aspect, the data used for the presentation of the 2D applicationis used to extract the respective user interface elements that areusually displayed to the user as 2D presentation. The interface elementsof the 2D world are translated into user interface features of a mixedreality world. The user provides input in the mixed reality world andthe input is mapped back into the 2D world to control the 2Dapplication.

Basically, a program is provided that uses an input system to simulateuser input to detect interactable interfaces from 2D screens withoutintegrating with existing software. In other words, an input system isused to simulate user input to detect the interactable interfaces from2D screens without integrating with the interactable source at asoftware level.

In an aspect, the use of augmented reality is applied to display screensand an interaction with those screens is provided. An example of thefield of use is healthcare, for example in the context of wearableaugmented reality devices and the application of other 2D applications.

According to an aspect, a device for controlling a 2D application in amixed reality application is provided with a 2D application interface, aprocessor, a mixed reality application interface and a controlinterface. The 2D application interface receives data representative ofa 2D application. The application comprises interactive elementsproviding commands for user control of the 2D application. The processoridentifies and extracts the interactive elements based on the datarepresentative of the 2D application. The processor also translates theinteractive elements to user interface features of a mixed realitydisplay application. The processor further generates data for presentingthe mixed reality display application and maps user input to commandsfor user control of the 2D application. The mixed reality applicationinterface provides the mixed reality display application to a user asmixed reality. The control interface receives input from the user forthe user interface features and provides the commands to the 2Dapplication in order to control the 2D application. Hence, augmentedreality interfaces are mapped to traditional 2D screen interfaces. Thisallows to integrate applications that only provided the interaction on amonitor with a mouse or the like to be operated in today's augmentedreality environment.

The integration of the 2D application into mixed or augmented reality issuitable for example for interventional guided therapy systems.

These and other aspects of the present invention will become apparentfrom and be elucidated with reference to the embodiments describedhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention will be described in thefollowing with reference to the following drawings:

FIG. 1 schematically shows an example of a device for controlling a 2Dapplication in a mixed reality application.

FIG. 2 schematically shows an example of a system for controlling a 2Dapplication in a mixed reality application.

FIG. 3 shows basic steps of an example of a method for controlling a 2Dapplication in a mixed reality application.

FIG. 4 illustrates a scenery in an operation room and the use of thesystem for controlling a 2D application in a mixed reality application.

FIG. 5 schematically shows a further setup for controlling a 2Dapplication in a mixed reality application.

FIG. 6 schematically shows another setup for controlling a 2Dapplication in a mixed reality application.

FIG. 7 illustrates an example of an enhanced augmented reality userinterface with colliders shown with frames in bright lines.

FIG. 8 illustrates an example of a translation of user interfaceelements into augmented reality user interface elements.

FIG. 9 illustrates an example of a 2D screen user interface in the leftpart, extracted button locations translated into augmented colliders inthe middle part and an augmented reality user interface design rule ofminimum button spacing requirement for eye gaze in the right part.

DETAILED DESCRIPTION OF EMBODIMENTS

Certain embodiments will now be described in greater details withreference to the accompanying drawings. In the following description,like drawing reference numerals are used for like elements, even indifferent drawings. The matters defined in the description, such asdetailed construction and elements, are provided to assist in acomprehensive understanding of the exemplary embodiments. Also,well-known functions or constructions are not described in detail sincethey would obscure the embodiments with unnecessary detail. Moreover,expressions such as “at least one of”, when preceding a list ofelements, modify the entire list of elements and do not modify theindividual elements of the list.

FIG. 1 schematically shows an example of a device 10 for controlling a2D application in a mixed reality application. The device 10 comprises a2D application interface 12, a processor 14, a mixed reality applicationinterface 16 and a control interface 18. The 2D application interface 12is configured to receive data representative of a 2D application. The 2Dapplication comprises interactive elements providing commands for usercontrol of the 2D application. The processor 14 is configured toidentify and extract the interactive elements based on the datarepresentative of the 2D application. The processor 14 is alsoconfigured to translate the interactive elements to user interfacefeatures of a mixed reality display application. The processor 14 isfurther configured to generate data for presenting the mixed realitydisplay application; and to map user input to commands for user controlof the 2D application. The mixed reality application interface 16 isconfigured to provide the mixed reality display application to a user asmixed reality. The control interface 18 is configured to receive inputfrom the user for the user interface features. The control interface 18is also configured to provide the commands to the 2D application inorder to control the 2D application.

The 2D application interface 12, the processor 14, the mixed realityapplication interface 16 and the control interface 18 may be providedintegrated in a common structure as indicated with frame 20. However,they may also be provided as separate components.

A first hashed arrow indicates the provision or supply from the 2Dapplication that is entering the 2D application interface 12.

A second hashed arrow indicates the output from the mixed realityapplication interface 16 to the mixed reality display application, whichis indicated as an option with a first hashed frame 26.

A second hashed frame 28 indicates the user input device as an option. Athird hashed arrow 30 indicates the provision from the user input deviceto the control interface 18.

The 2D application interface relates to an input interface for providingthe data of the 2D application that is used for presenting the 2Dapplication. The 2D application interface can also be referred to as 2Ddisplay application interface or 2D application interface input. The 2Dapplication interface can also be referred to as screen interface ormonitor interface.

The 2D application interface may be provided as an interface that takesscreen content from a native 2D application and sends it to theaugmented reality application. This can be implemented as aframe-grabber or via proprietary data interfaces.

The processor provides data processing. The processor can also bereferred to as computing arrangement or controller.

In example, the processor performs the following key tasks: Identifyinguser interaction elements in the native application screen, translatingthose user interaction elements into augmented reality user interactionelements having predefined colliders, and setting properties of thecolliders based on the augmented reality user interaction rules. As anoption, the processor also performs mapping user input towards augmentedreality user interaction elements into an input that can be fed back tothe native application.

The mixed reality application interface relates to an output interfacefor providing the data used for presenting the mixed reality displayapplication. The mixed reality application interface can also bereferred to as mixed reality display data supply or mixed realitypresentation output.

The control interface relates to an input interface for the input fromthe user for the user interface features. The control interface alsorelates to an output interface for the commands to control theapplication. The control interface thus provides the interface functionin two directions: From the user towards the mixed reality applicationinterface and from the mixed reality application (and this from theuser) to the 2D application.

The control interface, or user interface controller, may be provided asan application that is part of the augmented reality application thatextracts user interaction features from the native 2D application screenand translates them to augmented reality display user interactionfeatures. Similarly, the user interaction controller can map backaugmented reality display user interaction commands back to commands tothe native 2D application via the control interface.

The control interface may be provided as an interface that can inputuser commands back to the native 2D application. This can be implementedvia mouse control (position and button down/up), keyboard input, touchscreen input, voice input, etc. In an example, the mixed reality relatesto augmented reality that comprises a reality view that is augmentedwith additional information.

In an example, the mixed reality relates to augmented virtuality thatcomprises a virtual view that is augmented with additional information.

In an example, the mixed reality relates to virtual reality thatcomprises a virtual view.

The data representative of a 2D application can also be referred to asdata representative of a 2D graphical user interface of a 2Dapplication. The 2D application can also be referred to as nativeapplication or 2D legacy content.

In an example, the 2D application runs with traditional 2D screens andinterfaces, such as an echo-navigator, interventional ultrasound,hemodynamics, Azurion Image Review and 3rd party applications such asoptic coherence tomography, electrophysiological mapping and the like.

In an example, a 2D application is provided as an application that isdisplayed on a 2D display. However, for example, it is provided that the2D display also displays multi-dimensional data.

The mixed reality application, such as an augmented reality applicationor a virtual reality application may be provided as an application thatgenerates the content displayed on the augmented reality display. Thisapplication may reside on the augmented reality display itself, or on aseparate computing device that sends content to the display.

In an example, a native application sends images and receives user inputvia an interface with the augmented reality application. The augmentedreality application has a user interaction controller that isresponsible for extracting user interaction features from the nativeapplication screen and then translates them to augmented reality displayuser interaction features. The user interaction controller also mapsaugmented reality interactions back to commands to the nativeapplication via the control interface.

In an example, the device provides an analysis of the 2D application andits interaction possibilities and then a translation into gestures.

In an example, it is provided to take into account gestures of thecurrent context. For example, the context of the operation is providedas input to determine gestures relevant for user interaction or gesturesless appropriate for user interaction.

As an example, not further illustrated, it is provided that the 2Dapplication interface is configured to receive: i) display data forpresenting the 2D application on a monitor; or ii) a sequence of imagesof a camera acquiring image data of a presentation of the 2Dapplication.

As an example, not further illustrated, for the identifying andextracting, it is provided that the processor 14 is configured toprovide an interactive elements identification routine comprising atleast one of the group of: detecting interaction features based on imageprocessing of the 2D application; detecting changes when moving aposition indicator across the 2D application presentation; recordinglocations of interaction activity by the user in the 2D applicationpresentation; and providing training datasets comprising interactionactivity and recognizing interactive elements based on machine learningof the training datasets.

In order to create custom augmented reality user interaction elements,user interaction elements in the native application must be identified.

In an example, the native application sends information about its userinteraction elements to the augmented reality application.

In another example, the processor runs a user interaction identificationroutine for any native application it encounters. This initializationroutine can be run every time the processor sees a change in the inputscreen. This initialization routine can also be run on startup of theaugmented reality application and/or the native application. Thisinitialization routine can also be run periodically as a calibrationstep.

In a further example, the user interaction controller runs the userinteraction identification routine in real-time based on where the useris looking or interacting with a screen.

The user interaction or user interface identification routine can beimplemented in multiple ways:

In a first example, image processing is provided to detect button-likefeatures in the display content as provided by the 2D application. Forexample, edges, changes in color for buttons etc. are detected in theimage content. The image content refers to the presentation of thecontent on a screen. The image content can also refer to the image dataas provided by the 2D application for the presentation on a screen.

In a second example, a routine is provided that moves the mouse acrosspixels and looks for changes in the interface. The mouse movement can beprovided automatically. The changes refer to e.g. visual changes thatare common for indicating user interaction possibilities to the user.Again, this can be done on the presentation as shown on the monitor orbased on the data as provided by the application.

In a third example, a routine is provided that uses machine learning torecognize buttons based on training datasets that consist of clickswhere users interact with the interfaces.

In a fourth example, a routine is provided that records locations ofbutton clicks by the user over time and recalls the locations of thosebutton clicks with regards to the screen display.

FIG. 2 schematically shows an example of a system 100 for controlling a2D application in a mixed reality application. The system 100 comprisesan example of the device 10 for controlling a 2D application in a mixedreality application according to one of the examples described above andbelow. The system 100 also comprises a mixed reality presentationarrangement 102, which is data-connected to the device for controlling a2D application in a mixed reality application. The mixed realitypresentation arrangement 102 comprises a presentation device 104 that isconfigured to present the mixed reality display application to a user asmixed reality. The mixed reality presentation arrangement 104 furthercomprises a user input device 106.

As an example, 2D screens can be used in image guided therapy togetherwith augmented reality wearables which improves ergonomics andflexibility within the interventional suite. Staff within the room canmaintain to rely on the 2D screens, such as the Flexvision, in order toreceive information from systems in the room. However, by transferringtheir information presentation and interaction into the augmentedreality, these screens can be moved aside or can even be omitted suchthat they no longer take up space in the room and it is also no longerrequired that their position accounts for sterility and for the positionof multiple staff members and other equipment. Providing the screens inaugmented reality enables the staff members to work in a more flexibleenvironment as they are not restricted to viewing the screens at awkwardand suboptimal positions. The combination of the 2D screen content inthe augmented reality world thus facilitates the work for staff members. Also, the adding of multiple screens to the room can be avoided. Theseadvantageous effects are caused by the visualization of informationthrough virtual screens via an augmented reality wearable thatalleviates the cumbersome issues experiences so far.

The visualization of information through virtual screens via anaugmented reality wearable together with the feedback and interactionpossibility addresses the need for providing flexible screens forvisualization to the user for improving ergonomics and flexibility inthe interventional suite. The visualization of information throughvirtual screens via an augmented reality wearable together with thefeedback and interaction possibility further also addresses the need forinteraction with those streams of information. Examples for userinterfaces are touch screen modules, table side controller ortraditional interfaces in form of a mouse and keyboard.

In an example, not further shown, the mixed reality presentationcomprises a presentation of the 2D application.

FIG. 3 shows basic steps of an example of a method 200 for controlling a2D application in a mixed reality application. The method 200 comprisesthe following steps: In a first step 202, data representative of a 2Dapplication is received. The 2D graphical user interface comprisesinteractive elements providing commands for user control of the 2Dapplication. In a second step 204, the interactive elements areidentified and extracted based on the data representative of the 2Dapplication. In a third step 206, the interactive elements aretranslated to user interface features of a mixed reality displayapplication. In a fourth step 208, data for presenting the mixed realitydisplay application to a user is generating and provided as mixedreality. The mixed reality display application comprises a presentationof the 2D application. In a fifth step 210, input from the user for theuser interface features is received. In a sixth step 212, the user inputis mapped to commands for user control of the 2D application. In aseventh step 214, the commands are provided to the 2D application inorder to control the 2D application.

As an option, indicated in FIG. 3 with hashed lines, it is furtherprovided an 8 ^(th) step 216 of providing a presentation of the mixedreality display application comprising the user interface features. Theuser input is based on the presentation of the mixed reality displayapplication.

In an example, it is provided to move the interactions anywhere in spaceand it is not restricted to the interaction elements on the screen thatis being streamed to the headset.

In an example, not further shown in detail, the translating of theinteractive elements to the user interface features comprises creatingcolliders around the identified interactive elements. The collidersenable a number of 3D feedback features.

In an example, the data representative of the 2D application comprisesdisplay data for presenting the 2D application on a monitor.

In an example, the data representative of the 2D application is providedas a sequence of images of a camera acquiring image data of apresentation of the 2D application.

In an example, for the identifying and extracting, it is provided aninteractive elements identification routine comprising at least one ofthe group of:

-   -   detecting button-like features based on image processing of the        displayed 2D application;    -   moving a mouse-pointer or the like across the presentation of        the 2D application and detecting changes in the 2D graphical        user interface;    -   recording locations of button clicks by the user in the 2D        graphical user interface and recalling the locations of those        buttons with regard to the 2D display;    -   providing training datasets comprising clicks where a user        interacts with the interactive elements and recognizing        interactive elements based on machine learning of the training        datasets.

Augmented reality generally refers to when a live image stream issupplemented with additional computer-generated information. The liveimage stream can be via the eye, cameras, smart phones, tablets, etc.This image stream is augmented via display to the user that can be donevia glasses, contact lenses, projections or on the live image streamdevice itself (smart phone, tablet, etc.). The present examples can beapplied to any implementation of augmented reality that overlays virtualobjects on the real world. In principal, this can also be extended tovirtual reality. Examples for augmented reality devices or virtualreality devices include the Mircrosoft's Hololens, or devices from MagicLeap, Vusix, or nReal.

The processor, also referred to as user interaction controller, or userinterface controller, provides the translation of the interactiveelements of the 2D application to user interface features of the mixedreality display application.

As an example, colliders are created around the user interaction objectsin the screen. Those colliders can then be used to enable a number ofaugmented reality user interaction features. For example, the cursorstyle changes when over a user interaction element. Another example isproviding audio feedback to the user when hovering, clicking or touchinga user interaction element. A further example is providing visualfeedback to the user when they are hovering, clicking, or touching auser interaction element. An example is also to provide haptic feedbackto the user, for example via the augmented reality interface, indicatingthat they are interacting with a user interaction element.

A still further example is to magnify an interaction element like abutton when the user is interacting with the user interaction element.For example, it can also be provided to allow the user a direct touch 3Dinteraction with a 2D element, e.g. direct touch ‘press’ of a button.

Another example is to provide gaze control and to implement dwell buttonpress functionality. For eye gaze, the processor can provide a minimumspacing between elements to facilitate control by the user.

The processor thus translates user interfaces that were designed for a2D screen and interaction with a mouse and keyboard into a userinterface that can accommodate gesture, eye gaze, head interaction,voice commands, etc.

The processor can be provided with augmented reality user interfacedesign rules such as minimum spacing between colliders, which is helpfulfor eye-gaze based button selection, Another example for an augmentedreality user interface design rule is to provide depths of colliders,which can be provided, for example, for direct touch interaction 2Dcolliders that require a 3D depth to allow for interaction. As a furtherrule, minimum or maximum size of a button can be determined. Forexample, a button must be of a minimum size to enable someone to selectit with eye gaze or head pose. The minimum size is also dependent on howfar away the button is from the user. A further rule can be to providegravity wells, which are helpful in eye gaze, where a gravity well canmake it harder to look off a button to an adjacent button. This helps toretain gaze on a selected object and avoid flickering between objects.

An example is further to provide head pose control and tab button press.

The mapping relates to assign and translate user input towards augmentedreality user interaction elements into an input that can be fed back tothe native application. For example, if the native application can onlyaccept mouse position and state (up/down), then the processor must mapthe augmented reality user interaction into those simplifiedinteraction. As an example, mouse position can be controlled the user byHead pose ray projection onto 2D virtual screen and to move the mouse upor down, pinch gesture state can be used, or a voice command ‘mousedown’, ‘mouse up’ or ‘select’. Another example for mouse position is touse eye gaze ray projection onto 2D virtual screen. To provide dwell forthree seconds may be provided for further improvement. A still furtherexample is to use voice command mapped to pixel position. To move themouse up or down, voice command of button can be applied. A furtherexample is to provide a button collider and to move the mouse, collisionbetween index finger position and button collider is provided.

In a further option, augmented reality user interaction elements areprovided by generating and mapping voice commands to a specific pixellocation in the screen. This can be provided by parsing the ‘labels’that appear on mouse-over for some interfaces. This can also be providedby parsing the names of buttons as the appear on the interface. This canalso be provided by allowing the user to label an augmented reality userinteraction button, for example looking at it and saying: create voicelabel “pause” or through a separate offline augmented reality userinteraction tool that allows the user to delineate buttons and populatea list of voice commands for that button. This can also be provided byhaving an interface that can send the voice label-to-pixel mapping fromthe native application. This can also be provided by having an interfacethat can create a voice label to a sequence of repeated button presses.For example, in case of an image review screen where there is a nextframe button, a new voice label “skip five frames” infers pressing thenext frame button five times.

In an aspect, the data used for the presentation of the 2D applicationis used to extract the respective user interface elements that areusually displayed to the user as 2D presentation. The interface elementsof the 2D world are translated into user interface features of a mixedreality world. The user provides input in the mixed reality world andthe input is mapped back into the 2D world to control the 2Dapplication.

Basically, a program is provided that uses an input system to simulateuser input to detect interactable interfaces from 2D screens withoutintegrating with existing software. In other words, an input system isused to simulate user input to detect the interactable interfaces from2D screens without integrating with the interactable source at asoftware level.

In an aspect, the use of augmented reality is applied to display screensand an interaction with those screens is provided. An example of thefield of use is healthcare, for example in the context of wearableaugmented reality devices and the application of other 2D applications.

The integration of the 2D application into mixed or augmented reality issuitable for example for interventional guided therapy systems.

FIG. 4 illustrates a scenery 300 in an operation room and the use of thesystem for controlling a 2D application in a mixed reality application.A subject 302 is arranged on a subject support. An X-ray imaging system304 may be provided, such as a C-arm imaging system. A monitorarrangement 306 is provided in the background showing the content of atraditional 2D application with the option of an interface in form of amouse or the like. A first staff member is wearing a first head-mountedaugmented reality device 308. The first user is provided with thecontent of the 2D application in form of an augmented realitypresentation 310 and to interact and thus control the 2D applicationwith the augmented reality device. In an option, the monitor in thebackground is omitted. As an option, a second staff member is wearing asecond head-mounted augmented reality device 312. The second user isprovided with the same or different content of the 2D application inform of an augmented reality presentation 314 and to interact and thuscontrol the 2D application with the augmented reality device.

In an option, the mixed reality presentation arrangement is an augmentedreality presentation arrangement that comprises a wearable augmentedreality device. As an option, a head-mounted display like the firsthead-mounted augmented reality device 308 or the second head-mountedaugmented reality device 312 is provided configured to present the mixedreality display application as an overlay to the reality.

An example for an augmented reality display is a head-mounted displayworn by at least one person in the interventional suite. This couldalternatively be a see-through screen, such as a heads-up-display usingthe lead shield. This could also be similar to RealView holographicdisplay. A further option is a virtual reality display.

In an example, a HoloLens is provided to display a virtual screen. Theaugmented reality application that generates the content of the virtualscreen comprises a frame-grabber that grabs the screen output from thenative application via a DVI-out interface. The user interactioncontroller interprets the virtual screen from the frame-grabber andgenerates additional user interaction elements such as colliders anduser interaction rules that are specific to augmented reality. The userinteraction controller translates user input into the virtual screen andmaps it to mouse input to the native application.

In an option, provided in addition or alternatively, the mixed realitypresentation arrangement is a virtual reality presentation arrangementthat comprises a wearable virtual reality device. As an option, ahead-mounted display is provided configured to present the virtualreality display application.

In an option of the system, an example of the augmented realitypresentation arrangement is used by at least one staff member and anexample of the virtual reality presentation arrangement is used by leastone further staff member.

As an example, not further shown in detail, a monitor like the monitorarrangement 306 is provided configured to present the 2D application. Asan option, an interface is provided to forward user interaction data tothe 2D application. The user interaction data is provided by the userinput device of the mixed reality presentation arrangement.

In case of an augmented reality application, in an example it isprovided to present the mixed reality display application such that itis overlaid to the monitor showing the real 2D application.

In case of a virtual reality application, in an example it is providedto present a presentation of the monitor in the virtual world in thesame spatial position (within the context of the room shown in thevirtual presentation) as in reality.

As an example, at least one of the following group is provided for userinteraction in the mixed reality presentation arrangement: gesturerecognition comprising recognition of moved totems; voice commands; headtracking; eye tracking; and external devices comprising controllers,remotes and keyboards.

FIG. 5 schematically shows a further setup 400 for controlling a 2Dapplication in a mixed reality application. A computer-monitorarrangement 402 is indicated. A native application 404 is provided to berun on the computer-monitor arrangement 402. A head-mounted augmentedreality device 406 is indicated for running an augmented realityapplication 408. A user interaction (or user interface) controller 410is provided. A first arrow 412 indicates a screen interface and a secondarrow 414 indicates a control interface.

FIG. 6 schematically shows another setup 500 for controlling a 2Dapplication in a mixed reality application. A computer-monitorarrangement 502 is indicated. A native application 504 is provided to berun on the computer-monitor arrangement 502. A display video output 506is provided and a mouse input 508. Further, a head-mounted augmentedreality device 510 is indicated for running an augmented realityapplication 512. A user interaction (or user interface) controller 514is provided. Further, a frame grabber 516 is indicated and a virtualscreen 518. A first arrow 520 indicates a first interface from thedisplay video output 506 to the frame grabber 516; and a second arrow522 indicates a second interface from the to the user interactioncontroller 514 to the mouse input 508. A second, bi-direction interface524 is indicated between the user interaction controller 514 and thevirtual screen 518. A further arrow 526 indicates the interface betweenthe Fran grabber 516 and the virtual screen 518. A still further arrow528 indicates the interface between the frame grabber 516 and the userinteraction controller 514.

FIG. 7 illustrates an example of an enhanced augmented reality userinterface 600 with colliders 602, 604 and 606 shown with frames inbright lines for the respective user interface portions of the 2Dapplication.

FIG. 8 illustrates a user interface 700 with examples of a translationof user interface elements into augmented reality user interfaceelements 702, 704 and 706.

FIG. 9 illustrates an example of a 2D screen user interface 800 in theleft part, extracted button locations translated into augmentedcolliders in the middle part 800′ and an augmented reality userinterface design rule of minimum button spacing requirement for eye gazein the right part 800″.

As an example, not further illustrated, the processor is configured tocreate colliders around the identified interactive elements for thetranslating of the interactive elements to the user interface features.The colliders enable a number of 3D feedback features.

In an option, the processor is configured to provide gravity wells atidentified interactive elements to help a user to hold an eye gaze ontoa location of an interactive element.

Wearable augmented reality provides a unique set of interactions for theuser. By transferring the interaction with the 2D application into theaugmented reality (or mixed reality) it is possible to interact withlegacy 2D content. Although a screen is streamed into the augmentedreality application without accompanying information about the availableinteractions and user interface elements, the user can interact withthat screen, for example, to indirectly push a button. The transferprovides the knowledge of the location of interactors within theaugmented reality environment which increases the functionality andpractical usability of the system. As an example, eye trackinginteraction is enabled. As an improvement, gravity wells can be createdthat help a user to hold their gaze onto a specific location in order toselect or click on something by eye gaze. Based on the knowledge ofwhere the gravity wells should reside within the interface, this type offunctionality is also provided in an option.

Based on the identification and extraction of the user interfaceinteractors of the 2D application into the augmented realityapplication, 3rd party interfaces and interactors can be integrated intothe augmented reality system. The identification and extraction alsoensure that updates of the 2D application interfaces and all possibleconfigurations of that particular interface are also reflected in theaugmented reality application.

The identification and extraction provide a dynamic extraction ofinformation from 2D interfaces and to use that information to enhanceaugmented reality interactions. This enables a plurality of interactiontypes, e.g. eye gaze, and therefore improves the user experience,increases efficiency of interactions and reduces errors when interactingwith 2D systems.

The term “subject” may also be referred to as individual. The “subject”may further also be referred to as patient, although it is noted thatthis term does not indicate whether any illness or disease is actuallypresent with the subject.

In an example, a computer program or program element for controlling anapparatus according to one of the examples above is provided, whichprogram or program element, when being executed by a processing unit, isadapted to perform the method steps of one of the method examples above.In another exemplary embodiment of the present invention, a computerprogram or a computer program element is provided that is characterizedby being adapted to execute the method steps of the method according toone of the preceding embodiments, on an appropriate system.

The computer program element might therefore be stored on a computerunit or be distributed over more than one computer units, which mightalso be part of an embodiment of the present invention. This computingunit may be adapted to perform or induce a performing of the steps ofthe method described above. Moreover, it may be adapted to operate thecomponents of the above described apparatus. The computing unit can beadapted to operate automatically and/or to execute the orders of a user.A computer program may be loaded into a working memory of a dataprocessor. The data processor may thus be equipped to carry out themethod of the invention.

Aspects of the invention may be implemented in a computer programproduct, which may be a collection of computer program instructionsstored on a computer readable storage device which may be executed by acomputer. The instructions of the present invention may be in anyinterpretable or executable code mechanism, including but not limited toscripts, interpretable programs, dynamic link libraries (DLLs) or Javaclasses. The instructions can be provided as complete executableprograms, partial executable programs, as modifications to existingprograms (e.g. updates) or extensions for existing programs (e.g.plugins). Moreover, parts of the processing of the present invention maybe distributed over multiple computers or processors.

As discussed above, the processing unit, for instance a controller,implements the control method. The controller can be implemented innumerous ways, with software and/or hardware, to perform the variousfunctions required. A processor is one example of a controller whichemploys one or more microprocessors that may be programmed usingsoftware (e.g., microcode) to perform the required functions. Acontroller may however be implemented with or without employing aprocessor, and also may be implemented as a combination of dedicatedhardware to perform some functions and a processor (e.g., one or moreprogrammed microprocessors and associated circuitry) to perform otherfunctions.

Examples of controller components that may be employed in variousembodiments of the present disclosure include, but are not limited to,conventional microprocessors, application specific integrated circuits(ASICs), and field-programmable gate arrays (FPGAs).

This exemplary embodiment of the invention covers both, a computerprogram that right from the beginning uses the invention and a computerprogram that by means of an up-date turns an existing program into aprogram that uses the invention.

Further on, the computer program element might be able to provide allnecessary steps to fulfil the procedure of an exemplary embodiment ofthe method as described above.

According to a further exemplary embodiment of the present invention, acomputer readable medium, such as a CD-ROM, is presented wherein thecomputer readable medium has a computer program element stored on itwhich computer program element is described by the preceding section. Acomputer program may be stored and/or distributed on a suitable medium,such as an optical storage medium or a solid-state medium suppliedtogether with or as part of other hardware, but may also be distributedin other forms, such as via the internet or other wired or wirelesstelecommunication systems.

However, the computer program may also be presented over a network likethe World Wide Web and can be downloaded into the working memory of adata processor from such a network. According to a further exemplaryembodiment of the present invention, a medium for making a computerprogram element available for downloading is provided, which computerprogram element is arranged to perform a method according to one of thepreviously described embodiments of the invention.

It has to be noted that embodiments of the invention are described withreference to different subject matters. In particular, some embodimentsare described with reference to method type claims whereas otherembodiments are described with reference to the device type claims.However, a person skilled in the art will gather from the above and thefollowing description that, unless otherwise notified, in addition toany combination of features belonging to one type of subject matter alsoany combination between features relating to different subject mattersis considered to be disclosed with this application. However, allfeatures can be combined providing synergetic effects that are more thanthe simple summation of the features.

While the invention has been illustrated, and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive. Theinvention is not limited to the disclosed embodiments. Other variationsto the disclosed embodiments can be understood and effected by thoseskilled in the art in practicing a claimed invention, from a study ofthe drawings, the disclosure, and the dependent claims.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. A single processor or other unit may fulfil the functions ofseveral items re-cited in the claims. The mere fact that certainmeasures are re-cited in mutually different dependent claims does notindicate that a combination of these measures cannot be used toadvantage. Any reference signs in the claims should not be construed aslimiting the scope.

1. A device for controlling a 2D application in a mixed realityapplication, the device comprising: a 2D application interfaceconfigured to receive data representative of a 2D application, whereinthe 2D application comprises interactive elements providing commands foruser control of the 2D application; a processor configured to: identifyand extract the interactive elements based on the data representative ofthe 2D application, translate the interactive elements to user interfacefeatures of a mixed reality display application, and generate data forpresenting the mixed reality display application; and to map user inputto commands for user control of the 2D application; a mixed realityapplication interface configured to provide the mixed reality displayapplication to a user as mixed reality; and a control interfaceconfigured to receive input from the user for the user interfacefeatures and provide the commands to the 2D application in order tocontrol the 2D application.
 2. The device according to claim 1, whereinthe 2D application interface is configured to receive: display data forpresenting the 2D application on a monitor; or a sequence of images of acamera acquiring image data of a presentation of the 2D application. 3.The device according to claim 1, wherein, for the identifying andextracting, the processor is configured to provide an interactiveelements identification routine comprising at least one of: detectinginteraction features based on image processing of the 2D application;detecting changes when moving a position indicator across the 2Dapplication presentation; recording locations of interaction activity bythe user in the 2D application presentation; and providing trainingdatasets comprising interaction activity and recognizing interactiveelements based on machine learning of the training datasets.
 4. Thedevice according to claim 1, wherein the processor is configured tocreate colliders around the identified interactive elements for thetranslating of the interactive elements to the user interface features;and wherein the colliders enable a number of 3D feedback features.
 5. Asystem for controlling a 2D application in a mixed reality application,the system comprising: the device for controlling a 2D application in amixed reality application according to claim 1; and a mixed realitypresentation arrangement connected to the device for controlling a 2Dapplication in a mixed reality application; wherein the mixed realitypresentation arrangement comprises a presentation device that isconfigured to present the mixed reality display application to a user asmixed reality; and wherein the mixed reality presentation arrangementcomprises a user input device
 6. The system according to claim 5,wherein the mixed reality presentation comprises a presentation of the2D application.
 7. The system according to claim 5, wherein the mixedreality presentation arrangement is an augmented reality presentationarrangement that comprises a wearable augmented reality device; andwherein, a head-mounted display is provided configured to present themixed reality display application as an overlay to the reality.
 8. Thesystem according to claim 5, wherein the mixed reality presentationarrangement is a virtual reality presentation arrangement that comprisesa wearable virtual reality device; and wherein, a head-mounted displayis provided configured to present the virtual reality displayapplication.
 9. The system according to claim 5, wherein a monitor isprovided configured to present the 2D application; and wherein, aninterface is provided to forward user interaction data to the 2Dapplication; wherein the user interaction data is provided by the userinput device of the mixed reality presentation arrangement.
 10. Thesystem according to claim 5, wherein at least one of the following isprovided for user interaction in the mixed reality presentationarrangement: gesture recognition comprising recognition of moved totems;voice commands; head tracking; eye tracking; and external devicescomprising controllers, remotes and keyboards.
 11. A method forcontrolling a 2D application in a mixed reality application, the methodcomprising: receiving data representative of a 2D application, whereinthe 2D graphical user interface comprises interactive elements providingcommands for user control of the 2D application; identifying andextracting the interactive elements based on the data representative ofthe 2D application; translating the interactive elements to userinterface features of a mixed reality display application; generatingand providing data for presenting the mixed reality display applicationto a user as mixed reality, wherein the mixed reality displayapplication comprises a presentation of the 2D application; receivinginput from the user for the user interface features; mapping the userinput to commands for user control of the 2D application; and providingthe commands to the 2D application in order to control the 2Dapplication.
 12. The method according to claim 11, further comprising:providing a presentation of the mixed reality display applicationcomprising the user interface features; wherein the user input is basedon the presentation of the mixed reality display application.
 13. Themethod according to claim 11, wherein the translating of the interactiveelements to the user interface features comprises creating collidersaround the identified interactive elements; and wherein the collidersenable a number of 3D feedback features.
 14. A non-transitorycomputer-readable storage medium having stored a computer programcomprising instructions, which, when executed by a processor, cause theprocessor to: receive data representative of a 2D application, whereinthe 2D graphical user interface comprises interactive elements providingcommands for user control of the 2D application; identify and extractthe interactive elements based on the data representative of the 2Dapplication; translate the interactive elements to user interfacefeatures of a mixed reality display application; generate and providingdata for presenting the mixed reality display application to a user asmixed reality, wherein the mixed reality display application comprises apresentation of the 2D application; receive input from the user for theuser interface features; map the user input to commands for user controlof the 2D application; and provide the commands to the 2D application inorder to control the 2D application.
 15. (canceled)